{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <p style=\"text-align:center\">\n",
    "        <img alt=\"phoenix logo\" src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/phoenix-logo-light.svg\" width=\"200\"/>\n",
    "        <br>\n",
    "        <a href=\"https://arize.com/docs/phoenix/\">Docs</a>\n",
    "        |\n",
    "        <a href=\"https://github.com/Arize-ai/phoenix\">GitHub</a>\n",
    "        |\n",
    "        <a href=\"https://arize-ai.slack.com/join/shared_invite/zt-2w57bhem8-hq24MB6u7yE_ZF_ilOYSBw#/shared-invite/email\">Community</a>\n",
    "    </p>\n",
    "</center>\n",
    "<h1 align=\"center\">Evaluate RAG with LLM Evals</h1>\n",
    "\n",
    "In this tutorial we will look into building a RAG pipeline and evaluating it with Phoenix Evals.\n",
    "\n",
    "It has the the following sections:\n",
    "\n",
    "1. Understanding Retrieval Augmented Generation (RAG).\n",
    "2. Building RAG (with the help of a framework such as LlamaIndex).\n",
    "3. Evaluating RAG with Phoenix Evals."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieval Augmented Generation (RAG)\n",
    "\n",
    "LLMs are trained on vast datasets, but these will not include your specific data (things like company knowledge bases and documentation). Retrieval-Augmented Generation (RAG) addresses this by dynamically incorporating your data as context during the generation process. This is done not by altering the training data of the LLMs but by allowing the model to access and utilize your data in real-time to provide more tailored and contextually relevant responses.\n",
    "\n",
    "In RAG, your data is loaded and prepared for queries. This process is called indexing. User queries act on this index, which filters your data down to the most relevant context. This context and your query then are sent to the LLM along with a prompt, and the LLM provides a response.\n",
    "\n",
    "RAG is a critical component for building applications such a chatbots or agents and you will want to know RAG techniques on how to get data into your application.\n",
    "\n",
    "<img src=\"https://storage.googleapis.com/arize-phoenix-assets/assets/images/RAG_Pipeline.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stages within RAG\n",
    "\n",
    "There are five key stages within RAG, which will in turn be a part of any larger RAG application.\n",
    "\n",
    "- **Loading**: This refers to getting your data from where it lives - whether it's text files, PDFs, another website, a database or an API - into your pipeline.\n",
    "- **Indexing**: This means creating a data structure that allows for querying the data. For LLMs this nearly always means creating vector embeddings, numerical representations of the meaning of your data, as well as numerous other metadata strategies to make it easy to accurately find contextually relevant data.\n",
    "- **Storing**: Once your data is indexed, you will want to store your index, along with any other metadata, to avoid the need to re-index it.\n",
    "\n",
    "- **Querying**: For any given indexing strategy there are many ways you can utilize LLMs and data structures to query, including sub-queries, multi-step queries, and hybrid strategies. \n",
    "- **Evaluation**: A critical step in any pipeline is checking how effective it is relative to other strategies, or when you make changes. Evaluation provides objective measures on how accurate, faithful, and fast your responses to queries are.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build a RAG system \n",
    "\n",
    "Now that we have understood the stages of RAG, let's build a pipeline. We will use [LlamaIndex](https://www.llamaindex.ai/) for RAG and [Phoenix Evals](https://arize.com/docs/phoenix/llm-evals/llm-evals) for evaluation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install -qq \"arize-phoenix[evals,llama-index]\" \"arize-phoenix-client\" \"llama-index-llms-openai\" \"openai>=1\" gcsfs nest_asyncio 'httpx<0.28'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this tutorial we will be using OpenAI for creating synthetic data as well as for evaluation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from getpass import getpass\n",
    "\n",
    "if not (openai_api_key := os.getenv(\"OPENAI_API_KEY\")):\n",
    "    openai_api_key = getpass(\"🔑 Enter your OpenAI API key: \")\n",
    "os.environ[\"OPENAI_API_KEY\"] = openai_api_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from llama_index.core import SimpleDirectoryReader, VectorStoreIndex\n",
    "from llama_index.core.node_parser import SimpleNodeParser\n",
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "from phoenix.client import Client\n",
    "from phoenix.client.types.spans import SpanQuery  # pyright: ignore[reportUnusedImport]\n",
    "\n",
    "client = Client()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During this tutorial, we will capture all the data we need to evaluate our RAG pipeline using Phoenix Tracing. To enable this, simply start the phoenix application and instrument LlamaIndex."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enable Phoenix tracing via `LlamaIndexInstrumentor`. Phoenix uses OpenInference traces - an open-source standard for capturing and storing LLM application traces that enables LLM applications to seamlessly integrate with LLM observability solutions such as Phoenix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openinference.instrumentation.llama_index import LlamaIndexInstrumentor\n",
    "\n",
    "from phoenix.otel import register\n",
    "\n",
    "tracer_provider = register(project_name=\"phoenix-rag-llama-index\")\n",
    "LlamaIndexInstrumentor().instrument(skip_dep_check=True, tracer_provider=tracer_provider)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Data and Build an Index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use an [essay by Paul Graham](https://www.paulgraham.com/worked.html) to build our RAG pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tempfile\n",
    "from urllib.request import urlretrieve\n",
    "\n",
    "with tempfile.NamedTemporaryFile() as tf:\n",
    "    urlretrieve(\n",
    "        \"https://raw.githubusercontent.com/Arize-ai/phoenix-assets/main/data/paul_graham/paul_graham_essay.txt\",\n",
    "        tf.name,\n",
    "    )\n",
    "    documents = SimpleDirectoryReader(input_files=[tf.name]).load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define an LLM\n",
    "llm = OpenAI(model=\"gpt-4o\")\n",
    "\n",
    "# Build index with a chunk_size of 512\n",
    "node_parser = SimpleNodeParser.from_defaults(chunk_size=512)\n",
    "nodes = node_parser.get_nodes_from_documents(documents)\n",
    "vector_index = VectorStoreIndex(nodes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Build a QueryEngine and start querying."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_engine = vector_index.as_query_engine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_vector = query_engine.query(\"What did the author do growing up?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the response that you get from the query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response_vector.response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default LlamaIndex retrieves two similar nodes/ chunks. You can modify that in `vector_index.as_query_engine(similarity_top_k=k)`.\n",
    "\n",
    "Let's check the text in each of these retrieved nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First retrieved node\n",
    "response_vector.source_nodes[0].get_text()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Second retrieved node\n",
    "response_vector.source_nodes[1].get_text()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that we are using Phoenix Tracing to capture all the data we need to evaluate our RAG pipeline. You can view the traces in the phoenix application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the traces by directly pulling the spans from the phoenix session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spans_df = client.spans.get_spans_dataframe(project_name=\"phoenix-rag-llama-index\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spans_df[[\"name\", \"span_kind\", \"attributes.input.value\", \"attributes.retrieval.documents\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the traces have captured the documents that were retrieved by the query engine. This is nice because it means we can introspect the documents without having to keep track of them ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spans_with_docs_df = spans_df[spans_df[\"attributes.retrieval.documents\"].notnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spans_with_docs_df[[\"attributes.input.value\", \"attributes.retrieval.documents\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have built a RAG pipeline and also have instrumented it using Phoenix Tracing. We now need to evaluate it's performance. We can assess our RAG system/query engine using Phoenix's LLM Evals. Let's examine how to leverage these tools to quantify the quality of our retrieval-augmented generation system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "\n",
    "Evaluation should serve as the primary metric for assessing your RAG application. It determines whether the pipeline will produce accurate responses based on the data sources and range of queries.\n",
    "\n",
    "While it's beneficial to examine individual queries and responses, this approach is impractical as the volume of edge-cases and failures increases. Instead, it's more effective to establish a suite of metrics and automated evaluations. These tools can provide insights into overall system performance and can identify specific areas that may require scrutiny.\n",
    "\n",
    "In a RAG system, evaluation focuses on two critical aspects:\n",
    "\n",
    "- **Retrieval Evaluation**: To assess the accuracy and relevance of the documents that were retrieved\n",
    "- **Response Evaluation**: Measure the appropriateness of the response generated by the system when the context was provided."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate Question Context Pairs\n",
    "\n",
    "For the evaluation of a RAG system, it's essential to have queries that can fetch the correct context and subsequently generate an appropriate response.\n",
    "\n",
    "For this tutorial, let's use Phoenix's `llm_generate` to help us create the question-context pairs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's create a dataframe of all the document chunks that we have indexed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's construct a dataframe of just the documents that are in our index\n",
    "document_chunks_df = pd.DataFrame({\"text\": [node.get_text() for node in nodes]})\n",
    "document_chunks_df = document_chunks_df.sample(10, random_state=42)\n",
    "document_chunks_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the document chunks, let's prompt an LLM to generate us 3 questions per chunk. Note that you could manually solicit questions from your team or customers, but this is a quick and easy way to generate a large number of questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "generate_questions_template = \"\"\"\\\n",
    "Context information is below.\n",
    "\n",
    "---------------------\n",
    "{text}\n",
    "---------------------\n",
    "\n",
    "Given the context information and not prior knowledge.\n",
    "generate only questions based on the below query.\n",
    "\n",
    "You are a Teacher/ Professor. Your task is to setup \\\n",
    "3 questions for an upcoming \\\n",
    "quiz/examination. The questions should be diverse in nature \\\n",
    "across the document. Restrict the questions to the \\\n",
    "context information provided.\"\n",
    "\n",
    "Output the questions in JSON format with the keys question_1, question_2, question_3.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "from phoenix.evals import OpenAIModel, llm_generate\n",
    "\n",
    "\n",
    "def output_parser(response: str, index: int):\n",
    "    try:\n",
    "        return json.loads(response)\n",
    "    except json.JSONDecodeError as e:\n",
    "        return {\"__error__\": str(e)}\n",
    "\n",
    "\n",
    "questions_df = llm_generate(\n",
    "    dataframe=document_chunks_df,\n",
    "    template=generate_questions_template,\n",
    "    model=OpenAIModel(model=\"gpt-3.5-turbo\"),\n",
    "    output_parser=output_parser,\n",
    "    concurrency=20,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct a dataframe of the questions and the document chunks\n",
    "questions_with_document_chunk_df = pd.concat([questions_df, document_chunks_df], axis=1)\n",
    "questions_with_document_chunk_df = questions_with_document_chunk_df.melt(\n",
    "    id_vars=[\"text\"], value_name=\"question\"\n",
    ").drop(\"variable\", axis=1)\n",
    "# If the above step was interrupted, there might be questions missing. Let's run this to clean up the dataframe.\n",
    "questions_with_document_chunk_df = questions_with_document_chunk_df[\n",
    "    questions_with_document_chunk_df[\"question\"].notnull()\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The LLM has generated three questions per chunk. Let's take a quick look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions_with_document_chunk_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieval Evaluation\n",
    "\n",
    "We are now prepared to perform our retrieval evaluations. We will execute the queries we generated in the previous step and verify whether or not that the correct context is retrieved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# loop over the questions and generate the answers\n",
    "for _, row in questions_with_document_chunk_df.iterrows():\n",
    "    question = row[\"question\"]\n",
    "    response_vector = query_engine.query(question)\n",
    "    print(f\"Question: {question}\\nAnswer: {response_vector.response}\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have executed the queries, we can start validating whether or not the RAG system was able to retrieve the correct context. Let's extract all the retrieved documents from the traces logged to phoenix. (For an in-depth explanation of how to export trace data from the phoenix runtime, consult the [docs](https://arize.com/docs/phoenix/how-to/extract-data-from-spans))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openinference.semconv.trace import DocumentAttributes, SpanAttributes\n",
    "\n",
    "retrieved_documents_df = client.spans.get_spans_dataframe(\n",
    "    project_name=\"phoenix-rag-llama-index\",\n",
    "    query=(\n",
    "        SpanQuery()\n",
    "        .where(\"span_kind == 'RETRIEVER'\")\n",
    "        .select(\"trace_id\", SpanAttributes.INPUT_VALUE)\n",
    "        .explode(\n",
    "            SpanAttributes.RETRIEVAL_DOCUMENTS,\n",
    "            reference=DocumentAttributes.DOCUMENT_CONTENT,\n",
    "            document_score=DocumentAttributes.DOCUMENT_SCORE,\n",
    "        )\n",
    "    ),\n",
    ")\n",
    "retrieved_documents_df.rename(columns={\"input.value\": \"input\"}, inplace=True)\n",
    "retrieved_documents_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now use Phoenix's LLM Evals to evaluate the relevance of the retrieved documents with regards to the query. Note, we've turned on `explanations` which prompts the LLM to explain it's reasoning. This can be useful for debugging and for figuring out potential corrective actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "relevance_template = \"\"\"\n",
    "You are comparing a reference text to a question and trying to determine if the reference text\n",
    "contains information relevant to answering the question. Here is the data:\n",
    "    [BEGIN DATA]\n",
    "    ************\n",
    "    [Question]: {input}\n",
    "    ************\n",
    "    [Reference text]: {reference}\n",
    "    ************\n",
    "    [END DATA]\n",
    "Compare the Question above to the Reference text. You must determine whether the Reference text\n",
    "contains information that can answer the Question. Please focus on whether the very specific\n",
    "question can be answered by the information in the Reference text.\n",
    "Your response must be single word, either \"relevant\" or \"unrelated\",\n",
    "and should not contain any text or characters aside from that word.\n",
    "\"unrelated\" means that the reference text does not contain an answer to the Question.\n",
    "\"relevant\" means the reference text contains an answer to the Question.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openinference.instrumentation import suppress_tracing\n",
    "\n",
    "from phoenix.evals import (\n",
    "    async_evaluate_dataframe,\n",
    "    create_classifier,\n",
    ")\n",
    "from phoenix.evals.llm import LLM\n",
    "\n",
    "llm = LLM(provider=\"openai\", model=\"gpt-4o\")\n",
    "\n",
    "relevance_eval = create_classifier(\n",
    "    name=\"relevance\",\n",
    "    prompt_template=relevance_template,\n",
    "    llm=llm,\n",
    "    choices={\"relevant\": 1.0, \"unrelated\": 0.0},\n",
    ")\n",
    "with suppress_tracing():\n",
    "    retrieved_documents_relevance_df = await async_evaluate_dataframe(\n",
    "        dataframe=retrieved_documents_df, evaluators=[relevance_eval]\n",
    "    )\n",
    "retrieved_documents_relevance_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retrieved_documents_relevance_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now combine the documents with the relevance evaluations to compute retrieval metrics. These metrics will help us understand how well the RAG system is performing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents_with_relevance_df = pd.concat(\n",
    "    [retrieved_documents_df, retrieved_documents_relevance_df.add_prefix(\"eval_\")], axis=1\n",
    ")\n",
    "documents_with_relevance_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compute Normalized Discounted Cumulative Gain [NCDG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) at 2 for all our retrieval steps.  In information retrieval, this metric is often used to measure effectiveness of search engine algorithms and related applications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import ndcg_score\n",
    "\n",
    "\n",
    "def _compute_ndcg(df: pd.DataFrame, k: int):\n",
    "    \"\"\"Compute NDCG@k in the presence of missing values\"\"\"\n",
    "    n = max(2, len(df))\n",
    "    eval_scores = np.zeros(n)\n",
    "    doc_scores = np.zeros(n)\n",
    "    parsed_scores = df.eval_relevance_score.apply(\n",
    "        lambda x: json.loads(x)[\"score\"] if isinstance(x, str) else x\n",
    "    ).values\n",
    "    eval_scores[: len(df)] = parsed_scores\n",
    "    doc_scores[: len(df)] = df.document_score\n",
    "    try:\n",
    "        return ndcg_score([eval_scores], [doc_scores], k=k)\n",
    "    except ValueError:\n",
    "        return np.nan\n",
    "\n",
    "\n",
    "ndcg_at_2 = pd.DataFrame(\n",
    "    {\"score\": documents_with_relevance_df.groupby(\"context.span_id\").apply(_compute_ndcg, k=2)}\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ndcg_at_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also compute precision at 2 for all our retrieval steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "precision_at_2 = pd.DataFrame(\n",
    "    {\n",
    "        \"score\": documents_with_relevance_df.groupby(\"context.span_id\").apply(\n",
    "            lambda x: x.eval_relevance_score.apply(\n",
    "                lambda s: json.loads(s)[\"score\"] if isinstance(s, str) else s\n",
    "            )[:2].sum(skipna=False)\n",
    "            / 2\n",
    "        )\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "precision_at_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, let's compute whether or not a correct document was retrieved at all for each query (e.g. a hit)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hit = pd.DataFrame(\n",
    "    {\n",
    "        \"hit\": documents_with_relevance_df.groupby(\"context.span_id\").apply(\n",
    "            lambda x: x.eval_relevance_score.apply(\n",
    "                lambda s: json.loads(s)[\"score\"] if isinstance(s, str) else s\n",
    "            )[:2].sum(skipna=False)\n",
    "            > 0\n",
    "        )\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now view the results in a combined dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "retrievals_df = client.spans.get_spans_dataframe(\n",
    "    query=(\n",
    "        SpanQuery()\n",
    "        .where(\"span_kind == 'RETRIEVER' and input.value is not None\")\n",
    "        .select(\"input.value\")\n",
    "    )\n",
    ")\n",
    "rag_evaluation_dataframe = pd.concat(\n",
    "    [\n",
    "        retrievals_df[\"input.value\"],\n",
    "        ndcg_at_2.add_prefix(\"ncdg@2_\"),\n",
    "        precision_at_2.add_prefix(\"precision@2_\"),\n",
    "        hit,\n",
    "    ],\n",
    "    axis=1,\n",
    ")\n",
    "rag_evaluation_dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observations\n",
    "\n",
    "Let's now take our results and aggregate them to get a sense of how well our RAG system is performing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Aggregate the scores across the retrievals\n",
    "results = rag_evaluation_dataframe.mean(numeric_only=True)\n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see from the above numbers, our RAG system is not perfect, there are times when it fails to retrieve the correct context within the first two documents. At other times the correct context is included in the top 2 results but non-relevant information is also included in the context. This is an indication that we need to improve our retrieval strategy. One possible solution could be to increase the number of documents retrieved and then use a more sophisticated ranking strategy (such as a reranker) to select the correct context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now evaluated our RAG system's retrieval performance. Let's send these evaluations to Phoenix for visualization. By sending the evaluations to Phoenix, you will be able to view the evaluations alongside the traces that were captured earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phoenix.client.resources.spans import SpanAnnotationData\n",
    "\n",
    "ndcg_rows = ndcg_at_2.reset_index()[[\"context.span_id\", \"score\"]].dropna()\n",
    "prec_rows = precision_at_2.reset_index()[[\"context.span_id\", \"score\"]].dropna()\n",
    "\n",
    "ndcg_annotations: list[SpanAnnotationData] = [\n",
    "    SpanAnnotationData(\n",
    "        name=\"ndcg@2\",\n",
    "        span_id=str(row[\"context.span_id\"]),\n",
    "        annotator_kind=\"CODE\",\n",
    "        result={\"score\": float(row[\"score\"])},\n",
    "    )\n",
    "    for _, row in ndcg_rows.iterrows()\n",
    "]\n",
    "\n",
    "precision_annotations: list[SpanAnnotationData] = [\n",
    "    SpanAnnotationData(\n",
    "        name=\"precision@2\",\n",
    "        span_id=str(row[\"context.span_id\"]),\n",
    "        annotator_kind=\"CODE\",\n",
    "        result={\"score\": float(row[\"score\"])},\n",
    "    )\n",
    "    for _, row in prec_rows.iterrows()\n",
    "]\n",
    "\n",
    "client.spans.log_span_annotations(span_annotations=ndcg_annotations, sync=False)\n",
    "client.spans.log_span_annotations(span_annotations=precision_annotations, sync=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Response Evaluation\n",
    "\n",
    "The retrieval evaluations demonstrates that our RAG system is not perfect. However, it's possible that the LLM is able to generate the correct response even when the context is incorrect. Let's evaluate the responses generated by the LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phoenix.client.types.spans import SpanQuery\n",
    "\n",
    "qa_df = client.spans.get_spans_dataframe(\n",
    "    project_name=\"phoenix-rag-llama-index\",\n",
    "    query=(\n",
    "        SpanQuery()\n",
    "        .select(\"span_id\", \"input.value\", \"output.value\")\n",
    "        .where(\"parent_id is None\")\n",
    "        .with_index(\"trace_id\")\n",
    "    ),\n",
    ")\n",
    "\n",
    "docs_df = client.spans.get_spans_dataframe(\n",
    "    project_name=\"phoenix-rag-llama-index\",\n",
    "    query=(\n",
    "        SpanQuery()\n",
    "        .where(\"span_kind == 'RETRIEVER'\")\n",
    "        .concat(\"retrieval.documents\", reference=\"document.content\")\n",
    "        .with_index(\"trace_id\")\n",
    "    ),\n",
    ")\n",
    "\n",
    "if qa_df is None or qa_df.empty:\n",
    "    print(\"No spans found.\")\n",
    "    qa_with_reference_df = qa_df\n",
    "elif docs_df is None or docs_df.empty:\n",
    "    print(\"No retrieval documents found.\")\n",
    "    qa_with_reference_df = None\n",
    "else:\n",
    "    ref = docs_df[[\"reference\"]]\n",
    "    qa_with_reference_df = pd.concat([qa_df, ref], axis=1, join=\"inner\").set_index(\n",
    "        \"context.span_id\"\n",
    "    )\n",
    "\n",
    "qa_with_reference_df.rename(\n",
    "    columns={\"input.value\": \"input\", \"output.value\": \"output\"}, inplace=True\n",
    ")\n",
    "qa_with_reference_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a dataset of the question, context, and response (input, reference, and output), we now can measure how well the LLM is responding to the queries. For details on the QA correctness evaluation, see the [LLM Evals documentation](https://arize.com/docs/phoenix/llm-evals/running-pre-tested-evals/q-and-a-on-retrieved-data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hallucination_prompt = \"\"\"\n",
    "In this task, you will be presented with a query, a reference text and an answer. The answer is\n",
    "generated to the question based on the reference text. The answer may contain false information. You\n",
    "must use the reference text to determine if the answer to the question contains false information,\n",
    "if the answer is a hallucination of facts. Your objective is to determine whether the answer text\n",
    "contains factual information and is not a hallucination. A 'hallucination' refers to\n",
    "an answer that is not based on the reference text or assumes information that is not available in\n",
    "the reference text. Your response should be a single word: either \"factual\" or \"hallucinated\", and\n",
    "it should not include any other text or characters. \"hallucinated\" indicates that the answer\n",
    "provides factually inaccurate information to the query based on the reference text. \"factual\"\n",
    "indicates that the answer to the question is correct relative to the reference text, and does not\n",
    "contain made up information. Please read the query and reference text carefully before determining\n",
    "your response.\n",
    "\n",
    "    [BEGIN DATA]\n",
    "    ************\n",
    "    [Query]: {input}\n",
    "    ************\n",
    "    [Reference text]: {reference}\n",
    "    ************\n",
    "    [Answer]: {output}\n",
    "    ************\n",
    "    [END DATA]\n",
    "\n",
    "    Is the answer above factual or hallucinated based on the query and reference text?\n",
    "\"\"\"\n",
    "\n",
    "qa_correctness_prompt = \"\"\"\n",
    "You are given a question, an answer and reference text. You must determine whether the\n",
    "given answer correctly answers the question based on the reference text. Here is the data:\n",
    "    [BEGIN DATA]\n",
    "    ************\n",
    "    [Question]: {input}\n",
    "    ************\n",
    "    [Reference]: {reference}\n",
    "    ************\n",
    "    [Answer]: {output}\n",
    "    [END DATA]\n",
    "Please read the query, reference text and answer carefully, then write out in a step by step manner\n",
    "an EXPLANATION to show how to determine if the answer is \"correct\" or \"incorrect\". Avoid simply\n",
    "stating the correct answer at the outset. Your response LABEL must be a single word, either\n",
    "\"correct\" or \"incorrect\", and should not contain any text or characters aside from that word.\n",
    "\"correct\" means that the question is correctly and fully answered by the answer.\n",
    "\"incorrect\" means that the question is not correctly or only partially answered by the\n",
    "answer.\n",
    "\n",
    "Example response:\n",
    "************\n",
    "EXPLANATION: An explanation of your reasoning for why the label is \"correct\" or \"incorrect\"\n",
    "LABEL: \"correct\" or \"incorrect\"\n",
    "************\n",
    "\n",
    "EXPLANATION:\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = LLM(provider=\"openai\", model=\"gpt-4\")\n",
    "\n",
    "hallucination_eval = create_classifier(\n",
    "    name=\"hallucination\",\n",
    "    prompt_template=hallucination_prompt,\n",
    "    llm=llm,\n",
    "    choices={\"factual\": 1.0, \"hallucinated\": 0.0},\n",
    ")\n",
    "qa_correctness_eval = create_classifier(\n",
    "    name=\"qa_correctness\",\n",
    "    prompt_template=qa_correctness_prompt,\n",
    "    llm=llm,\n",
    "    choices={\"correct\": 1.0, \"incorrect\": 0.0},\n",
    ")\n",
    "with suppress_tracing():\n",
    "    results_df = await async_evaluate_dataframe(\n",
    "        dataframe=qa_with_reference_df,\n",
    "        evaluators=[hallucination_eval, qa_correctness_eval],\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Observations\n",
    "\n",
    "Let's now take our results and aggregate them to get a sense of how well the LLM is answering the questions given the context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_correctness_eval = results_df.qa_correctness_score.apply(\n",
    "    lambda x: json.loads(x)[\"score\"] if isinstance(x, str) else x\n",
    ").mean()\n",
    "\n",
    "print(f\"Q&A Correctness Score: {qa_correctness_eval}\")\n",
    "qa_correctness_eval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hallucination_eval = results_df.hallucination_score.apply(\n",
    "    lambda x: json.loads(x)[\"score\"] if isinstance(x, str) else x\n",
    ").mean()\n",
    "\n",
    "print(f\"Hallucination Mean Score: {hallucination_eval}\")\n",
    "hallucination_eval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we have evaluated our RAG system's QA performance and Hallucinations performance, let's send these evaluations to Phoenix for visualization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from phoenix.evals.utils import to_annotation_dataframe\n",
    "\n",
    "relevancy_eval_df = to_annotation_dataframe(dataframe=results_df)\n",
    "\n",
    "client.annotations.log_span_annotations_dataframe(\n",
    "    dataframe=relevancy_eval_df,\n",
    "    annotator_kind=\"LLM\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have sent all our evaluations to Phoenix. Let's go to the Phoenix application and view the results! Since we've sent all the evals to Phoenix, we can analyze the results together to make a determination on whether or not poor retrieval or irrelevant context has an effect on the LLM's ability to generate the correct response."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "We have explored how to build and evaluate a RAG pipeline using LlamaIndex and Phoenix, with a specific focus on evaluating the retrieval system and generated responses within the pipelines. \n",
    "\n",
    "Phoenix offers a variety of other evaluations that can be used to assess the performance of your LLM Application. For more details, see the [LLM Evals](https://arize.com/docs/phoenix/llm-evals/llm-evals) documentation."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
